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Introduction. The current study is aimed at understanding the mecha¬ 
nisms governing online rating systems such as Yelp, IMDB etc. The so- 
called referral systems have become so popular over the years one can only 
wonder how they work and if they do at all. 

Online ratings are based on users’ impressions and opinions they form after 
using a certain product or service. Hence, the first unknown in the equation 
is the mechanism under which a user decides to purchase a certain good 
or use some service. We identify two factors that come into play in such a 
choice; a subjective factor and an objective factor. The subjective factor is 
the impression the user forms on the product or business; the gut feeling 
that tells him whether it is worth his time and money. The gut feeling 
is not always reliable; how many times have you had a good impression 
of a restaurant only to find out that it is not entirely to your taste? The 
reliability of the customer’s gut feeling turns out to be an important factor in 
the model. The objective factor relates to the quality of the business assessed 
independently of users’ tastes or preferences. Often times, the online rating 
of a business is taken as a proxy for its objective ?quality rating?. The 
underlying assumption is that by averaging ratings across users of varying 
tastes and preferences, dependencies upon said factors are eliminated. 

It is essential to note the importance of subjectivity in a user’s choice. We 
stress the fact that a business or product is not simply good or bad; rather, 
it is good or bad from a certain user’s perspective. Businesses that are per¬ 
ceived as good by a majority of users are labeled as “high quality” businesses 
and businesses that are perceived as good by a minority of users are labeled 
as “low quality” businesses. A natural question to ask is whether inherently 
high-quality businesses can ever be labeled as low quality through online 
rating systems. 

Next we examine the mechanism under which online ratings are formed. 
A user chooses to try a certain business or not by a rational evaluation of 
the subjective and objective factors under a user-agnostic utility function. 
Upon trying a business the user leaves a rating that is a reflection of his own 
subjective perception of its quality. Given that we take the online rating 
to be the objective factor in the decision making process, a user’s action is 
affected by the actions of all the users who have tried the business in the 
past, hence the notion of a sequential move game. 
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Under this model, it is clear that the ?objective? online rating is sequen¬ 
tially updated through users’ subjective ratings. Thus at its early stages, 
a business is at the mercy of the subjectivity of its customers. In theory, 
some businesses could die simply because they were first approached by the 
wrong customers; others could be overrated by similar mechanisms. In the 
long run however, one would expect the online rating to converge to an ob¬ 
jective measure of a business?s quality. We seek to determine how often and 
under what conditions this is the case. 

Notation. We write 


Bern(p) or X Unif [0i i] 

to express that a random variable X is distributed according to a Bernoulli 
distribution with parameter p £ [0,1] or according to a uniform distribution 
over the interval [0,1], respectively. Furthermore if P is a property we write 

t rpj f 1 if P holds > 

^ [ 0 otherwise. 

With this notation in mind, we are now equipped to introduce more formally 
the general format of what we call a rating game. 

Rating Game. Suppose consumers i £ N = {1, 2, 3,... } must make some 
decision sequentially according their order. The decision might be whether 
to adopt a product or service of a firm whose overall quality is given by 
a £ [0,1]. In our consideration we take the firm to be a restaurant, and the 
restaurant quality a is taken to represent the portion of the total population 
who will enjoy their experience at the restaurant (i.e. the objective factor). 
Each consumer i will independently either like or dislike the restaurant ac¬ 
cording to Vi ~ Bern(a), but i will only learn of this value if she goes to the 
restaurant. Instead, consumer i knows the value of a private signal Xi (the 
subjective factor), which is equal to Vi with probability p. That is, 

Xi ~ Bern (pvi + (1 - p)( 1 - v^), 

so if v t = 1 then x t ~ Bern(p) is accurate to Vi with probability p; and if 
Vi = 0 then Xi ~ Bern(l — p) is also accurate to Vi with probability p. In 
a sense, then, p represents how well consumers know what they like. The 
value of p is common knowledge, so we may assume that the private signals 
are on average accurate at least half the time p > 0.5 (indeed if p < 0.5 then 
each consumer takes her new private signal to be 1 — x,, which is accurate 
to Vi on average 1 — p > 0.5 of the time). 

Each consumer i £ N must decide in order whether to attend the restaurant, 
and she knows (1) her own private signal Xj and (2) the average value of the 
Vj ’s for consumers j < i who have attended the restaurant already. If TV) is 
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the set of all j < i who attended the restaurant, we denote this average 


Vi 


1 

W\ 


2 

j&Ni 


and when Ni is empty we take i>i = \ (this initialization is an important 
assumption, and we consider later generalizations of the rating game where 
the restaurant can choose the initialization). We call the value V{ the rating. 
As a summary of this game, we have 

• Players - consumers i £ N. 

• Moves - each j £ N decides, in order, whether to attend a restaurant. 

• Information - each * G N knows her private signal x t and the current 
rating v t . 

• Payoffs - if i does not attend the restaurant, her payoff is 0; other¬ 
wise, her payoff is 2 (vi — |), i.e. if she does like it, her payoff is 1 
and if she doesn’t, her payoff is —1. 


Solution. Individuals do not observe their type. Hence, in order to decide 
whether to go to the restaurant or not, each individual will proceed by 
maximizing over types his ex-ante expected payoff given some signal x t . 
Recall that each individual has a type v t e {0,1}, observes a private signal 
Xi e {0,1} and choose an action a* £ {0,1}, where a, = 1 indicates that 
he goes to the restaurant and a* = 0 indicates that he does not. The ex- 
ante expected payoff from choosing action a, for individual i is denoted as 
E [u(ai, Uj)|xj], where u{ai,Vi) is the utility individual i gets by choosing 
action ai when his type is v t . The optimal choice of action for individual i, 
denoted a*, is given by: 


a* = argmax E[u(aj, 

Cli 

= argmax ^ u(a Xl q)p(vi = q\xi) 

ai 96 ( 0 , 1 } 

= argmax ——- V' u(oj, q)p{vi = q, x,) 
ai P[Xi) 96(0,1} 

= argmax ^ u(o*, q)p(vi = q)p(xi\vi = q) 

“* 96(0,1} 

1 {^ 96 ( 0 , 1 } vi 1 ’ q)p( v i = q)p( x i = M v i = q) > X^ri} q)p( v i = q)p( x i 

1 {X96{o,i} vi 1 ’q)pi v i = q)pi x i = = q)> X96(o,i} q)p( v i = q)p( x i 

1 {—(1 — a)(l — p) + ap > 0 x (1 — a)(l — p) + 0 x ap} if cc* = 1 
1 {—(1 — a)p + a(l — p) > 0 x (1 — a)p + 0 x a(l — p)} if Xi = 0 

1 {—1 + a + p — ap + ap > 0} if a:* = 1 
1 {—p + ap + a ~ ap > 0} if x* = 0 

1 {a > 1 — p] if Xi = 1 
1 {a > p) if Xi = 0 


ik = q) 
ok = q) 


if Xi = 1 
otherwise. 
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More generally, the derivation of the optimal action choice doesn’t require 
an explicit functional form for the utility function u(a,i,Vi). The only as¬ 
sumption made is that u(a t , 0) + u(a,i, 1) = 0. In other words, the payoff 
from going to a low quality restaurant is compensated by the payoff from 
going to a high quality restaurant. Similarly, the payoff from not going to 
a low quality restaurant is compensated by the payoff from not going to a 
high quality restaurant. 

On the other hand, the optimal actions do make a lot of sense. First, indi¬ 
viduals require more compelling evidence to go to a restaurant when they 
receive a negative private signal (a* = 1 when Xi = 0 if a > p) as opposed 
to a positive private signal (a* = 1 when = 1 if a > 1 — p). Moreover, we 
note that the “objective quality parameter” a plays the role of an adjust¬ 
ment. Naturally, the private signal xi is noisy and doesn’t always coincide 
with Vi- As a result, the individual doesn’t simply follow the private signal 
Xi but factors in the information carried by a. Precisely, if individual i is 
more likely to enjoy the restaurant than his signal is to be right (a > p), 
the optimal action specifies that he should go to the restaurant regardless 
of his private signal a* = 1 (V Xi G {0,1}). Intuitively, if (a > p > 0.5) and 
X{ = 1, then everything indicates that he will enjoy the venue and the ratio¬ 
nal choice can only be to dine there. If on the other hand (a > p) and x t = 0, 
then he must choose whether to follow his private signal or the objective rat¬ 
ing. Given that the private signal is more likely to be wrong than he is to 
not enjoy the venue, the rational choice is to dine at the venue. By analogy, 
if the individual is less likely to enjoy the restaurant than his signal is to 
be wrong then even with a positive private signal (xi = 0), the individual 
chooses to avoid the venue (a* = 0). By contrast, when the objective quality 
parameter doesn’t fall in any of these two edge cases (p > a > 1 — p), the 
individual simply follows his private signal: 

* / 1 {a > 1 - p} = 1 if Xi = 1 _^ * 

| 1 {a > p} = 0 if Xi = 0 a * Xl 

Thus, the noisy signal x t is trusted unless there is compelling evidence from 
a that sways the balance of probabilities in one direction or the other. 

Finally, we note that individuals do not observe the objective quality pa¬ 
rameter a and are forced to estimate it somehow. The essence of our work 
lies in the fact that individuals are assumed to use the average online rating 
Vi as an estimator of a. The fact that h* is not necessarily a consistent esti¬ 
mator of a is not an issue; it is precisely what makes the results interesting. 


Comparison. We consider the similarities and differences with the rating 
game and other games in which individuals must combine noisy private 
information with inferred information from other players. A motivating 
scenario from probability is usually described as follows: 
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A prison warden has been given explicit directions to reveal to three prison¬ 
ers on death row- Alice, Bob, and Charlie—precisely the information that 
one of them (chosen at random) had been pardoned and that the remaining 
two of them will be executed in the coming week. None of the prisoners can 
communicate to each other, but they each speak with the warden regularly. 
Prisoner A asks the warden to reveal who will be executed. The warden, 
careful not to reveal to A any information on her own fate, reveals that B 
will be executed. 

The point is that A may mistakenly reason that, now since one of A or C 
must be executed, her chances of being executed have risen from | to The 
warden was clever to realize, however, that the probability of A’s execution 
is independent of the news of B’s execution, so the probability remains at 
Jj, which one shows using Bayes’ rule. A similar tradeoff between having 
tangible information and being able to make use of it is realized gradually 
in the rating game. At first, player 1 has no information of other players’ 
preferences and must make her decision based solely on her private signal x\ 
and the initial rating v\ = suppose player 1 does visit the restaurant, in 
which case player 2 knows x-2 and V2 = v\. So player 2 knows exactly whether 
or not player 1 liked the restaurant, but this tells him very little more about 
his own preferences. Since v, varies from person to person, player 2 wants 
a more aggregate indication of how well consumers liked the restaurant. As 
j 6 N increases and more players have contributed to the rating, those who 
have decided to try the restaurant may have become a self-selective pool, 
relying both on their private signal and the current rating. So while Vi for 
small i depends greatly on the inclinations of a few consumers, r>i as i gets 
larger may become a substantially biased estimator of a; as we shall see, it 
depends in significant ways on how well individuals know their own tastes. 

For example, in the case where p = 1 , all the consumers know precisely 
whether they like the restaurant because Xi = Uj. So consumers do not need 
to know the rating in order to decide: they just follow their private signals. 
It is interesting to note, however, that in this special case we have the rating 
converges Vi -A 1 = p as i — > oo for any a > 0. 

An important related model depicts an information cascade. In one example, 
an employer might find a job applicant very promising, but knowing that 
the applicant has been rejected for similar jobs in the past may cause the 
employer to ignore his own impression and reject the applicant anyway; in 
another example, a stock value can become overinflated when individuals 
ignore their personal evaluations of its quality because they see the stock 
value spike. In this model the users all get the same value Vi = v £ {0, 1 } 
from the restaurant, so we might think of a rating game with a £ {0,1} 
chosen according to a Bern(^). That way n, = a for every consumer i £ N 
but they simply don’t know whether the restaurant is good or bad. Instead of 
getting a rating from previous consumers, in an information cascade model, 
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consumer i knows whether each previous j < i decided to try the restaurant. 
Each consumer i also has their private signal Xi which still depends on p 
in the same way. Supposing that indeed V{ = a = 1 (individuals will like 
the restaurant if they go) we showed (in an assignment) that in this model, 
by the time individual 3 makes her decision, already with probability p 2 is 
everyone willing to ignore their private signal and go to the restaurant (a 
‘correct cascade’) and with probability (1 — p) 2 is everyone willing to ignore 
their private signal and avoid the restaurant (an ‘incorrect cascade’). We 
made two essential alterations when coming up with the rating game : Vi 
depends on the individual i in our model, and each individual cannot see 
the decisions of the previous, only the rating they leave. Both changes are 
made partially with the hope that it becomes more feasible to prevent either 
cascade and partially as a nice context to study the rating system. 

Experiments and Results. Fixing the strategy described in the ‘solution’ 
section, it begins to make sense to ask about simulating this game, since some 
quantities like the rating u* converge as i gets very large. One quantity we 
might study using simulated data is the average rating, to see if it converges 
to what we expect. Fixing p = 0.8, we plot the average rating V{ versus i 
for various a. 



Figure 1. Rating m versus consumers i for various a. 

When a > p we have Vi —> a as we expected. We interpret this to mean 
that whenever enough people will like the restaurant, it does not matter how 
good people are at knowing their own preferences because a critical mass 
of people who try the restaurant and leave good reviews will keep people 
coming and trying the food. In other words, the ratings are so compelling 
that every individual chooses to go to the restaurant. As a result, the portion 
of the population who goes reflects the overall population well. Hence the 
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proportion of people who go and like it converges to the true proportion 
of people who like it. When a G [0.5, p] we appear to have Vi -A p. We 
interpret this to mean that when the quality is ‘so-so’ (more accurately, 
when it is liked by a fair amount of the population), the average rating 
tends to be higher than the quality a itself. This suggests that a moderate 
quality restaurant can attain a degree of self-selectivity in its consumer base. 
Individuals are wary of the so-so rating and decide to trust their private 
signal. As a result, only individuals with a positive private signal decide 
to go to the restaurant. Among them, a larger fraction likes the restaurant 
than in the overall population, hence Vi > a. When a € [1 — p, 0.5) it is 
harder to tell what happens: v t converges to some value between a and p, 
increasing with respect to a. It can be shown using Bayes’ rule that when 
Vi converges, it approaches 

• ( oip \ 

\ap+ (1 -a)(l - p) ) 

In other words, customer selection through private signals weighs the rating 
upwards for medium quality restaurants. The rating however is not bound 
to increase forever. If the rating becomes too large, everyone wants to go to 
the restaurant regardless of their private signal. It is the end of self-selection 
and everyone starts rating the business. The ratings become more and more 
reflective of the true business quality and start decreasing. Soon enough, 
the ratings become so-so again and only individuals with positive signals 
go to the venue. Intuitively, the process should lead the rating to converge 
hence the above mathematical expression. 

Finally, we observe that some of the runs displayed in Figure 1 show a 
flat rating at around 0.2. These correspond to runs where unlucky busi¬ 
nesses were visited in early stages by negative customers. Their ratings 
subsequently took a hit, and the evidence showing that they are low-quality 
restaurants became overwhelming for anyone to knock on their door ever 
again. 


More formally, it has been noted in the ‘solution’ section that if the rating 
falls below 1 — p, consumers will stop going to the restaurant regardless of 
their private signal. We refer to this as “business death.” So in a given 
simulation a business can die fairly quickly. Consider the following recursive 
formula for the rating: 


Vi +1 = 


Vi 

IM+il+i 


if i + 1 doesn’t go 
if i + 1 goes 


where |W+i| is the number of consumers who went to the restaurant before 
i + 1. If enough consumers who dislike the restaurant go, it may end up 
the case that Vi falls below 1 — p, in which case the business dies, because 
no consumer wants to go to a restaurant with Uj < 1 — p. We estimate the 
business death after 1000 consumers by averaging over 1000 simulated trials 
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for over 5000 (p, a) pairs. We reason that since Vi can only change very little 
at a time after i > 1000 that our estimate is a very good approximation for 
the total probability of business death. 


Probability of business death a/ter 1000 consumers 



0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.0 

rho 


Figure 2. Probability of business death versus ( a,p ). 

This figure has some nice properties. For a fixed value of p, the probability 
of business death decreases for increasing a. When a = 0, the restaurant 
fails no matter the value of p because anyone who goes will leave a negative 
rating. Similarly when a = 1, the restaurant cannot fail because anyone 
who goes loves it. When p = 1, everyone knows what they want, so any 
restaurant with a > 0 can survive. Contrast this case with p = 0.5, when 
the private signal Xi is essentially useless. In this case, only restaurants with 
a > 0.5 can survive the rating system. More generally the worse individuals 
are at knowing what they like, the more likely they are to go to a restaurant 
they dislike, the more likely a restaurant is to be unlucky, receive the wrong 
customers and die. 

One feature of the plot which is hard to explain is the vertical discontinuities 
at | and The more refined we make this plot, the more pronounced those 
discontinuities become. The smooth transition between the regime where 
every restaurant dies and every restaurant survives is perhaps the most 
interesting feature of this graph. It is clear from this plot that if a restaurant 
got to decide a or p, they should prefer a = 1, but if they have to have a < 1 
they want p to be as large as possible. In other words however small a is, a 
business can avoid death by making sure that the right customers and only 
the right customers enter through the door. This effect can be sustained 
by maintaining an average rating and relying on customers to follow very 
accurate private signals. 
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Recognizing that the majority of consumers are operating within this par¬ 
adigm, the keen (say level k + 1) consumer would want an idea of how 
accurate the rating is to the true quality. For this reason we plot the rating 
bias , defined as the average rating minus the quality a. 




0.85 
rho 

Figure 3. Rating bias uiooo — a for various (p, a). 

When a is very high or very low, the rating bias is nearly zero. This is 
a consequence of the fact that when everybody likes the restaurant, it will 
always get good ratings, and when nobody likes the restaurant, it will always 
get bad ratings. More generally, the rating bias seems to depend more on p 
than on a. When p is large (so that everyone knows what they want), the 
lower quality restaurants get a high upward bias, i.e. their rating is higher 
than their quality. To an outsider who does not know what they want, in a 
world where everyone leaves reviews only for the places they like, the rating 
system is useless. Towards the periphery of the semi-circle on the right 
seems to be the ‘sweet spot’ where the rating system favors restaurants, 
and conversely on the left semi-circle we have the rating system asserting 
a highly negative bias. Around where a = 0.5 and p = 0.5 the downward 
bias makes it so restaurants have ratings significantly lower than their actual 
quality. Since business death is so prominent in this region of the plot, this 
is what probably kills their ratings. 

We wanted to come up with one more mechanism to mitigate these rating 
biases, particularly the negative bias in moderate quality restaurants when 
people have low p. So in Figure 4 we plot the rating bias when only half of 
the population reads the reviews. The motivation for this is that business 
death is cause by customers reading bad reviews. In contrast, no business 
(at least for which a > 0) can totally die when some consumers only follow 
their private signal. 



Average rating bias e/ter 
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Average rating bias e/ter 1000 consumers 
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Figure 4. Rating bias uiooo — ct for various (p,ct), ^ readers 

Introducing this sub-population sort of levelled out the bias in the average 
rating. To be more precise, the bias is almost entirely eliminated in the 
regime where p < 0.75. As hypothesized, the previously observed negative 
bias was due to business death, which has become less likely. In the regime 
where p > 0.75 the bias is still present, positive and large for small a. 

Conclusions. Through mathematical analysis and simulations, on-line rat¬ 
ings and their impact on businesses were characterized through two param¬ 
eters: the inherent and objective restaurant quality factor, a, and the ac¬ 
curacy of a customer’s gut feeling about a business, p. In particular, it was 
shown that on-line ratings were seldom accurate mainly because of the low 
or high accuracy in customers’ gut feelings. 

When customers are very uncertain with regards to their preferences, they 
tend to purchase products or services that they do not like and leave bad 
reviews. If they so happen to be one of the first customers to rate the 
business, the bad review is a condemnation to death. On the other hand, 
accurate gut feelings tend to advantage medium-quality businesses through 
self-selection. In general, no matter how bad a business or product is, it will 
still be appreciated by a fraction of the population. By relying on accurate 
gut feelings, customers who decide to purchase the low-quality product will 
most often end up enjoying it and positively rating it. 

Given that this effect is more pronounced for low quality businesses, it is 
very likely the case that on-line referral systems help low-quality businesses 
survive. Furthermore, if only a fraction of consumers read on-line reviews, 
the business death risk is reduced for low-quality businesses and on-line 
ratings further shift the scales in favor of low-quality businesses. 
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Relevant Code. 

function[rate, death] = rating(a, p, N) 

7"/ Given alpha-rho values a,p and a number of customers N, 
77 Returns the rating after N customers have opportunity to 
7 Assume first person follows private signal 
rate = 0.5; 
t = 1; 

r = 0; "/number of ratings so far 

7, User never goes when alpha smaller than 1-rho 
while (rate > 1-p) && (t <= N) 

"/User likes the restaurant with prob. alpha 
v = rand < a; 

"/Message is correct with probability rho 
x = rand < (p*v + (1-p)*(1-v)); 

7 User always goes if alpha greater than rho 
7 or x = 1 and alpha greater than 1 - rho 
if (rate > p) || x 

rate = (r*rate + v) / (r + 1); 
r = r + 1; 

end 

t = t+1; 

end 

"/ indicates whether business died before 
"/ the first N customers decided 
death = (t < N); 

end 

77 Script to generate business death plots 
alpha = 0:.01:1; I = length(alpha); 
rho = 0.50000101:1; J = length(rho); 

S = 10~3; N = 10~3; M = zeros(I,J); 
for i=l:I 

for j = 1:J 

p = rho(j); a = alpha(i); 

B = zeros(l,N); 
for n=l:N 

[~,B(n)] = rating(a, p, S); 

end 

M(i,j) = sum(B)/N; 

end 

end 

imagesc(flipud(M)); 



